ACCOUNTABILITY 


WAKE COUNTY a DATA, RESEARCH & 


PUBLIC SCHOOL SYSTEM 


Multi-Tiered System of Support, 
2016-17 to 2017-18 


Abstract 


The Multi-Tiered System of Support (MTSS) framework, which uses a systems approach to promote 
school improvement and support all students in improving academics and behavior using data-based 
problem-solving, is a key part of the Wake County Public Schools System (WCPSS) Strategic Plan. As of 
2017-18, MTSS schools were at the initial stages of MTSS implementation and showed no significant 
improvement in Grades 3 and 8 reading proficiency, overall achievement, or Education Value-Added 
Assessment System (EVAAS) growth ratings that could be attributed to MTSS implementation. 
Suspension rates and eligibility for special education in high incidence categories were not yet a focus 
for the practices or professional learning and were also not positively affected by MTSS. However, a 
higher percentage of teachers in treatment than control elementary schools reported that MTSS 
positively affected student achievement in their schools. Additionally, MTSS had a positive effect on 
Hispanic/Latino students’ 2017-18 graduation rates and on reduction of their special education 
eligibility rates at elementary schools. The initial results suggest that MTSS should remain a district 
priority with an ongoing leadership support, expanded focus on tiered supports and on the behavioral 
component of MTSS. 
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This evaluation report describes the progress made by the Wake County Public School System 
(WCPSS) towards meeting the long-term goals for Multi-Tiered System of Support (MTSS) as of 
2017-18. WCPSS started exploration and installation of MTSS in 2014-15 by offering professional 
learning to WCPSS Senior Leadership and to a group of 28 schools (MTSS Cohort 1). To assign the 
rest of the district schools to MTSS cohorts for professional learning, a pool of 88 schools was 
created out of the schools that did not receive professional learning. Half of those were randomly 
assigned to Cohort 2 which received professional learning and started implementation in 2016-17, 
while the other half was assigned to Cohort 4 which started their professional learning and 
implementation two years later. The remaining schools became Cohort 3. While all four Cohorts 
received professional learning, this MTSS report is focused specifically on the schools randomly 
assigned to Cohort 2 and to their comparison, Cohort 4. 


What is MTSS? 


MTSS is a multi-tiered framework which promotes school improvement through engaging, 
research-based academic and behavioral practices. MTSS uses a systems approach to supporting 
students in improving both academics and behavior using data-based problem-solving to maximize 
growth for all (North Carolina Department of Public Instruction, n.d.). 


MTSS addresses persistent academic and behavioral disparities among student subgroups, 
including gaps in student outcomes and the disproportionality in suspension rates. It strives to 
reduce the predictability of academic outcomes by helping students overcome early academic 
difficulties. The framework is also designed to address behavioral disparities that often present 
themselves in the form of high suspension rates for certain student subgroups. The MTSS 
framework is incorporated into WCPSS district policies and is a key part of the district’s current 


Vision2020 Strategic Plan. 


The MTSS framework encompasses six major components: 
e Leadership 

e Communication and Collaboration 

e Building Capacity/Infrastructure for Implementation 

e = Three Tiers Instruction/Intervention 

e Data-Based Problem Solving 

e =Data/Evaluation 


MTSS emphasizes continuous school improvement through strengthening school leadership, 
building school capacity and infrastructure, and promoting data-evaluation processes (Self- 
Assessment of MTSS Implementation, 2015). According to the professional learning feedback and 
the Cohort 2 Self-Assessment of MTSS (SAM) ratings provided by the Instructional Leadership 
Teams (ILTs) at the schools, the implementation of most of these components in 2017-18 was still 
in progress. The primary emphasis of the professional learning and implementation to date has 
been on academics and particularly on Tier | (core instruction). Implementation of other 
components of MTSS took place with varying degrees of fidelity, but have not yet been the primary 
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focus of MTSS in WCPSS. As such, this study is focused more heavily on the components of MTSS 
which the district has addressed in these first years of implementation. 


Results 


This report addresses the student outcomes comprising the long-term MTSS goals. The outcomes 
presented in this report are based on two years of MTSS implementation by Cohort 2 and their 
comparison to Cohort 4 schools that did not begin implementation until 2018-19. Research shows 
that it generally takes three to five years for a school staff implementing a new approach with 
fidelity before the student outcomes begin to show some effect (Fixsen, Naoom., Blase, Friedman, 
& Wallace, 2005), even when all components of that system are in place. Therefore, all results 
reflecting the current state of MTSS implementation in WCPSS should be considered preliminary. 


This report includes descriptive analyses for all student outcomes, as well as additional analyses to 
control for possible effects of prior student achievement, demographic characteristics, and school 
effects (see Appendix B for more detail). These additional analyses provide more confidence in 
attributing any possible change in the outcomes of interest to the implementation of MTSS rather 
than other outside factors. The 2017-18 outcomes did not reveal much in the way of changes in 
student outcomes related to MTSS implementation, with the exception of higher graduation rates 
for Hispanic/Latino students, and lower eligibility for special education support in high incidence 
disabilities for Hispanic/Latino students in elementary schools. Also, a higher percentage of 
elementary school teachers In Cohort 2 than in Cohort 4 felt implementation of MTSS had 
positively affected student outcomes in their school. 


Recommendations 


Recommendations focus on ways to further strengthen MTSS implementation through central 
services support and inter-departmental collaboration: 


e The MTSS framework should remain a district priority. It is still being developed and is still at 
initial stages of implementation on some dimensions. The district needs to maintain consistent 
leadership team and inter-departmental support and collaboration at all levels to provide 
effective assistance to school leadership teams, school counselors, and intervention specialists. 


e Early Warning System (EWS) data should be used more consistently to identify the schools 
with more barriers to implementation of MTSS and offer those schools additional 
implementation support. 


e Additional emphasis on the behavioral component of MTSS is needed through district-level 
cross-departmental collaboration to improve implementation fidelity for behaviorally focused 
MITSS strategies; the adoption of a K-12 behavior assessment system will be helpful to further 
this work. 
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e School Improvement Plans (SIPs) have been restructured as a result of a year-long effort of the 
district’s MTSS/SIP leadership team. District leadership should continue to monitor and 
strengthen implementation of the new school improvement process to maintain alignment 
with MTSS. 


e Strengthen application of standardized procedures for tiered student support in Tiers II and III 
and more appropriate identification of students eligible for special education support in high 
incidence disability categories. Continued support and learning are needed to help schools 
prepare for the 2020 Specific Learning Disability (SLD) Eligibility policy. 


e Since the MTSS logic model should be a living document, the district leadership should 
annually revisit and update the model to reflect implementation progress. 
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Background 


MTSS is a complex multi-tiered framework which uses a systems approach to promoting school 
improvement and supporting all students in improving both academics and behavior using data- 
based problem-solving to maximize growth for all. 


Following the extensive collaboration with the Florida’s Problem Solving/Response to Intervention 
Project team in 2014-15, WCPSS worked in partnership with the North Carolina Department of 
Public Instruction (NCDPI) and the National Implementation Research Network (NIRN) to 
strengthen implementation of MTSS. NCDPI supported the WCPSS MTSS leadership team in 
completing the District Capacity Assessment (DCA) in 2016 to measure the district’s overall 
capacity to support MTSS implementation and identified the standardized tools for implementing 
tiered support. NIRN introduced Implementation Science concepts to the MTSS leadership team 
and shared their generalized walkthrough tool in the fall of 2017. 


As of 2017-18, in some respects, WCPSS was ahead of the state of North Carolina in terms of MTSS 
implementation: 


e The district’s MTSS leadership team was the first in the state to plan out the implementation 
process by charting out the strategies to address the district areas of need and by specifying 
the short-term, intermediate, and long-term goals. 


e MTSS language was embedded in several district policies, including Student Promotion and 
Accountability and Student Behavior/Code of Student Conduct. 


e The Early Warning System (EWS) was set up to identify groups of students in need of support 
and their areas of need. The EWS is currently used by the district schools, while NCDPI has 
delayed the official rollout of the statewide version of the system until the end of the 2018-19 
school year. 


e A Standard Treatment Protocol for the use of the replicable and empirically validated 
intervention for all students with similar academic or behavioral needs is being developed and 
introduced to the elementary and middle schools. 


e A Tiered Behavioral Resource Guide is being developed by a collaborative central office group 
to help with tiered behavioral support. 


e Professional learning was offered to the entire district with a four-year rollout. 


At the same time, it is important to note that according to SAM ratings completed by school 
leadership teams, the district is still at the initial stages of implementation. According to 
Implementation Science theory (see Figure A1 in Appendix A), any innovation goes through 
several stages of implementation, which include exploration, installation, initial implementation, 
and full implementation, before reaching sustainability (The National Implementation Research 
Network, n.d.). The district is yet to move to full implementation. 
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Program Goals 


A pathway of change for MTSS is presented below. It is a concise version of the MTSS logic model 
designed in 2015-16, which delineated the needs, specified the short-term, intermediate, and 
long-term outcomes for MTSS and described the strategies for achieving those outcomes. 
According to the model, MTSS aims to overcome disparities in both achievement and behavior 
that exist among student subgroups, so that the district can reach the 2020 WCPSS goal of 95% 


graduation rate. 


Figure 1 
Pathway of Change 


Effort: Multi-Tiered System of Support (MTSS) 
Need: Disparities in achievement and behavior exist among student subgroups that create gaps in 
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Data Sources 


This report is primarily focused on implementation and student outcomes to date for MTSS 
Cohorts 2 and 4. Some potentially useful implementation data sources were not available or 
represented components of MTSS which have yet to be put in place. As such, Table 1 details the 
main components of MTSS and data sources that were examined for the study. 


Table 1 

MTSS Components and Data Sources Used for Implementation Outcomes 
MTSS Components Data Sources Available /Examined 
Leadership training attendance & training 

feedback, SAM 

Communication and Collaboration beliefs survey results, SAM 
Building Capacity/Infrastructure for SAM, coaching survey results 
Implementation 
Three Tiers Instruction/Intervention SAM 
Data-Based Problem Solving (DPBS) SAM 
Data/Evaluation initial student outcomes, SAM 


Several key student performance indicators were examined in 2017-18 to determine whether 
there was any impact associated with the implementation of the MTSS framework. Various data 
sources available to Data, Research, and Accountability Department (DRA) were used for this 
evaluation. Those included student rosters containing student-level demographics information 
and End of Grade/End of Course (EOG/EOC) results, suspension data, and special education 
eligibility data. School-level data with EVAAS ratings, performance composites, and graduation 
results by school were also examined. 


In 2017-18, while Cohort 2 was in the second year of implementation of MTSS, Cohort 4 schools 
were in their first year of training and were at the exploration stage of implementation of MTSS. 
Because schools were assigned randomly to Cohorts 2 and 4, differences in outcomes that might 
be found between the cohorts can be directly attributed to differences in implementation of 
MTSS rather than other factors. The key student outcome questions guiding the report and data 
sources used to address those questions are presented in Table 2. 


MTSS 2016-17 to 2017-18 December 2018 


Table 2 
Key Outcome Questions and Data Sources 


Research Question Data Source 


Did Grade 3 and 8 reading proficiency for Cohort 2 improve Grades 3 and 8 EOG Reading 
significantly by the end of year 2 of MTSS implementation? scores 

Did overall achievement measured by performance composite | Schoolwide achievement results 
improve for Cohort 2 by the end of year 2 of MTSS (all tests/subjects) 


Implementation? 
Did Cohort 2 schools improve their EVAAS ratings compared to | NCDPI EVAAS growth ratings 
2015-16 and compared to Cohort 4 schools? 
Did graduation rates at Cohort 2 schools increase in 2017-18 Four-year graduation rates 
overall and for subgroups of students compared to the 
previous two years and compared to Cohort 4 schools? 
Did suspension rates and percentage of students suspendedin | Suspension rates 
2017-18 decrease compared to 2015-16 for Cohort 2 more 
than in Cohort 4? 

Did the percentages of students with high incidence disabilities | Special education eligibility rates 
in Cohort 2 become more representative of the population 
from 2015-16 to 2017-18 and compared to Cohort 4? 


Study Design 


This evaluation is based on an experimental design where one of each pair of matched schools 
was randonlly assigned to either the treatment (Cohort 2) or the control (Cohort 4) group. The 
randomized staggered rollout meant that the treatment cohort of schools (Cohort 2) started their 
professional learning in 2015-16 and implementation in 2016-17 and continued implementation 
in 2017-18, while the control cohort (Cohort 4) started their training two years later, in 2017-18. 
The report offers the analyses of the 2017-18 Cohort 2 outcomes and their comparisons to Cohort 
4 schools. 


Table 3 
Nature of the Data Provided and Valid Uses 


Research Design Conclusions that Can be Drawn 


M Experimental We can conclude that the program or policy caused changes in 
outcomes because the research design used random 
assignment. 

O Quasi-Experimental We can reasonably conclude that the program or policy caused 


changes in outcomes because an appropriate comparison 
strategy was used. 


VI Descriptive These designs provide outcome data for the program or policy, 
but differences cannot be attributed directly to it due to lack of a 


M Quantitative comparative control group. 
OQ Qualitative 


Sources: List, Sadoff, & Wagner (2011) and What Works Clearinghouse (2014). 
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Implementation 


As of spring 2017-18, year 2 of the MTSS implementation for Cohort 2, two of the eight 
implementation goals specified in the MTSS logic model were fully met. Four goals were mostly 
met, and two goals were still in progress (Bulgakov-Cooke, Lenard, & Singh, 2018). As of October 
2018, the School Improvement Plans (SIPs) goal was fully met, and some evidence of 
implementation of MTSS was reflected in 2018 WCPSS Teacher Survey. 


Table 4 
Implementation Outcomes as of Spring and Fall 2018 
Implementation Outcomes Spring 2018 Fall 2018 
Professional Development 
Deliver four days of PL with 90% attendance kk & N/A 
80% a staff indicate increased knowledge and potential XY N/A 
effectiveness 
Coaching 
90% of ILT responses reflect regular coaching and KY N/A 
technical support impacting practices 
Beliefs 
Certified school staff will have more positive beliefs (80% 
of beliefs items become more positive) in spring of year 2 Kw kw N/A 
than in the fall of year 1 
SAM 
Kk * 


By the third administration of SAM (2017-18), 90% of 


N/A 
schools increase their school domain averages second 


admininstration 


Average rating of SAM items for 90% of schools will reflect 


expanded and increased use of the Team-Initiated wk N/A 
Problem Solving model (second administration) 

SIPs 
An integration of MTSS components will be reflected in Available in kk *& 
the SIP key processes or action steps October 2018 

Core Instruction 
Evidence of implementation of MTSS in core instruction Available in tess’ 
(Tier 1) summer 2018 

Goal Status: fully met mostly met partially met not met 

KKK kK Kektk boxe xd 
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Training: Professional learning attendance records from 2018 showed that all four cohorts 
received professional learning with at least 90% attendance. However, due to the staggered 
implementation timeline, the amount of training differed by cohort (see Figure 2). While Cohort 2 
received three years of professional learning, Cohort 4 had only received one year. According to 
the daily training feedback survey, almost 80% of Cohort 2 respondents indicated increased 
knowledge and potential effectiveness. 


Figure 2 
MTSS Cohort Training Timeline 


Cohort 1 


ee Cohort 2 
Cohort 3 
Cohort 1 Cohort 4 
Cohort 2 
Cohort 3 
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Cohort 2 
District 
& Leadership 
Cohort 1 


Coaching: As of 2017-18, based on the coaching survey results, the coaching goal of 90% of 
Instructional Leadership Team (ILT) responses reflecting regular coaching and technical support 
impacting practices was mostly met. Communication skills and ability to provide timely feedback 
were rated by ILT members the highest (86% to 90% agreement), followed by the ability to build 
trust and facilitate consensus. At least 83% of ILT members were satisfied with coaching support, 
and 78% considered the coach effective in helping their school with implementation of MTSS and 
action planning. The ability to provide technical assistance, support with decision-making and 
problem solving were rated the lowest (58% to 71%). 


Beliefs: The goal for certified school staff to have more positive beliefs (for 80% of Beliefs Survey 
items to become more positive) in the spring of 2017 than in the fall of 2015 was mostly met. 
Cohort 2 teacher beliefs significantly improved in three of the four survey areas from 2015 to 
2017. As another point of comparison, baseline (pre-implementation) beliefs ratings were 
actually higher for Cohort 4 than Cohort 2, but after two years of implementation for Cohort 2, 
those gaps largely disappeared. 
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Figure 3 
2015 through 2017 Beliefs Survey Results 
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Table 5 
2015 through 2017 Beliefs Survey Results 


Gahorts Cohort 2 Difference a pre hea Cohort 2 vs. Cohort 4 
2015 to 2017 ‘ 2017 Comparison 
Comparison 


Factor 1: Academic Ability and Performance of Students with Disabilities 


Cohort 2 Not Significant 

Cohort 4 & 2 4>2 

Cohort 4 &2 4>2 
Factor 2: Data-Based Decision Making 

Cohort 2 Significant 

Cohort 4&2 4>2 

Cohort 4 &2 Not Significant 
Factor 3: Functions of Core and Supplemental Instruction 

Cohort 2 Significant 

Cohort 4&2 4>2 

Cohort 4 &2 Not Significant 
Factor 4: Cultural Proficiency 

Cohort 2 Significant 

Cohort 4&2 4>2 

Cohort 4&2 Not Significant 


SAM: The goal for 90% of schools increasing their school implementation domain averages by the 
third administration of SAM was measured using the second administration of SAM available in 
2017, and showed significant increases in all SAM domains. The SAM scale, which ranges from 
zero to three (0 = not implementing, 1 = emerging/developing, 2 = operationalizing, and 3 = 
optimizing), showed an average implementation score of 1.6 overall (Figure 4). Cohort 2 schools, 
with an average overall rating of 1.6, were still moving from “emerging/developing” to 
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“operationalizing” levels and have not yet reached the “optimizing” levels of implementation. 
Cohort 4 schools, as expected, rated themselves much lower than Cohort 2 schools in all domains 
in 2017-18 as they were still in the beginning stages of the work. 


Figure 4 
Summary of the Self-Assessment of MTSS Implementation Domain Averages 
for Cohorts 2 and 4, 2017 
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20 
1.0 1.9 16 
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TIPS model: The goal for the average rating of SAM items for 90% of schools to reflect expanded 
and increased use of the Team-Initiated Problem Solving model (as indicated by the DPBS score) 
was mostly met by spring 2018. SAM showed that 71% of schools expanded and increased the 
use of the TIPS model during the second administration. 


School Improvement Plans: The goal of the integration of MTSS components into SIPs was met by 
the end of October 2018. 


Evidence of implementation of MTSS in core instruction (Tier |) data were collected through the 
annual WCPSS teacher survey due to lack of a “look-for” observational data. In the survey, 
teachers were asked how familiar they were with MTSS and about their perceptions of the effects 
of the MTSS instructional framework used at their school on students’ academics and behavior. 
Significantly higher percentages of elementary school teachers in Cohort 2 than Cohort 4 stated 
that they were familiar with the MTSS instructional framework used at their school. Significantly 
higher percentages of elementary school teachers in Cohort 2 than Cohort 4 believed that the 
implementation of MTSS positively affected academic achievement. There was no difference 
between the cohorts in teacher perceptions of the effect of MTSS on student behavior. 
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Figure 5 
Selected 2018 WCPSS Teacher Survey Responses for Cohorts 2 and 4 
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Student Outcomes 


This section of the report describes the 2017-18 student outcomes for Cohort 2. Those included 
reading proficiency in grades 3 and 8, schoolwide achievement (i.e., performance composites), 
school EVAAS ratings, and graduation rates. This section also examines suspension rates for 
subgroups of students and eligibility rates for special education support in high incidence 
disability categories for subgroups of students. 


Most of the expected long-term student outcome goals were not yet met as of 2017-18. 
However, two exceptions were observed: a significant graduation rate increase in 2017-18 
compared to 2014-15 for Hispanic/Latino students in Cohort 2 and a decrease in special 
education eligibility rates in high incidence disabilities for Hispanic students at elementary 
schools. This is not unexpected at this point, because professional learning had not yet focused 
on all components of MTSS. 


Table 6 
Student Outcomes as of Fall 2018 


Long-Term Student Outcomes 2017-18 
Grade 3 and 8 Reading 
Proficiency in reading in grades 3 and 8 increases WI WW 


Performance Composite 


Performance composite for student subgroups increases WW WW 
EVAAS Ratings 

Percentage of schools with improved EVAAS ratings increases WHT Ww 
Graduation 

Graduation rates increase > aka Xe 
Suspension 


The number of suspension incidents decline (overall and by ethnicity) 


The number of students suspended decline (overall and by ethnicity) ei es 


Eligibility for SPED support in high incidence disability categories 


The proportion of students eligible for special education support in high 


incidence disability categories becomes more representative of the > at ate 
population 
Goal Status: fully met mostly met partially met not met 
KKK KKK Kkk yk ak a a 
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Did Grade 3 and 8 Reading proficiency improve significantly by the end of year 2 of 
MTSS implementation? 


In Grade 3, Reading EOG results showed a slight decrease in grade 
level proficiency rates from 2014-15 to 2017-18 overall and by ethnic 
subgroup. 


In Grade 8, the overall percentages of students demonstrating 
proficiency on Reading EOG increased slightly for both cohorts, and 
were flat for most subgroups. 


Analyses showed no significant difference between the cohorts. 


Grade 3 


An analysis of Grade 3 Reading EOG results showed a slightly decreasing trend in the percentages 
of students demonstrating grade-level proficiency from 2014-15 to 2017-18 in both Cohorts 2 
and 4. All student ethnic subgroups showed similar trends, except Hispanic/Latino students in 
Cohort 2, whose proficiency rates remained flat. Cohort 2 LEP and SWD students’ proficiency 
rates were also flat. Black/African American students in both cohorts showed the largest 
percentage point decrease (see Figure 6). In addition to the descriptive data, multilevel analyses 
controlling for other school- and student-level factors, also did not show any statistically 
significant difference between Cohorts 2 and 4. 
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Cohorts 2 and 4: Percentages of Grade 3 Students 
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Grade 8 


In Grade 8, the overall percentage of students demonstrating grade level proficiency in Reading 
started out slightly higher and remained slightly higher for Cohort 2 than Cohort 4. Both cohorts 
showed a slight improvement between 2014-15 and 2017-18, with flat rates for most subgroups. 
Multilevel analyses controlling for school- and student-level factors, did not show any statistically 
significant difference between Cohorts 2 and 4. 
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Figure 7 
Cohorts 2 and 4: Percentages of Grade 8 Students 
Demonstrating Grade Level Proficiency in Reading 
Cohort 2 Grade 8 Reading Proficiency Cohort 4 Grade 8 Reading Proficiency 
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Data Source: 2017-18 Student Rosters 


Did overall achievement measured by performance composite increase significantly by 
the end of year 2 of MTSS implementation? 


There was no significant difference between the years 2014-15 to 
2017-18, on performance composites for either Cohort 2 or Cohort 
4 schools overall or across subgroups. There was no significant 
difference between the cohorts. 
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Performance composite is a statewide accountability measure that calculates the percent of 
scores of students in a school that are at or above proficiency. For our analyses, we used the 
percentage of test scores at level 3 or higher. There was no significant difference between the 
years 2014-15 and 2017-18, or between cohorts, on performance composite scores overall or 
across subgroups. Multilevel analyses similarly showed no significant difference between the 


cohorts. 


Figure 8 


Cohorts 2 and 4: 2014-15 to 2017-18 Percentages of Test Scores Proficient by Subgroup 
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Data Source: 2017-18 Student Rosters 
Note: All results are reported at grade level proficiency 
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Did Cohort 2 schools improve their EVAAS ratings compared to 2014-15 and compared 
to Cohort 4 schools? 


The 2017-18 EVAAS ratings for Cohort 2 schools remained at 2014-15 
levels. There was no difference between the cohorts overall or for 
subgroups in 2017-18. 


Prior to MTSS implementation, Cohort 2 schools’ 2015 EVAAS ratings were 4.6 percentage 
points higher than those of Cohort 4. Both Cohorts first increased their EVAAS ratings, and both 
decreased in 2018. Multilevel analyses showed no significant difference between the Cohorts. 


Percent of Schools with Positive 


EVAAS Ratings 


Figure 9 
Cohorts 2 and 4 EVAAS Ratings from 2015 to 2017 
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Did graduation rates at Cohort 2 schools increase overall and for subgroups of 
students compared to 2015-16 and to Cohort 4? 
_——————— a ee eee 


In 2017-18, Cohort 2 had graduation rates similar to 2015-16 and similar 
to Cohort 4. However, Cohort 2 demonstrated significantly higher 
graduation rates for Hispanic/Latino students than those in Cohort 4. 


After an initial five-point graduation rate growth from 2014-15 to 2016-17, Cohort 2 schools 
showed a decrease in 2017-18 and overall graduation rates that were similar to Cohort 4. 


Figure 10 
Cohorts 2 and 4 2014-15 to 2017-18 Graduation Rates 
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A closer look at subgroups showed a large 22 percentage point graduation rate increase 
for Cohort 2 LEP students (see Figure 11). However, this was related to the new LEP 
identification procedures developed by NCDPI. In 2017-18, Cohort 2 demonstrated four 
percentage points higher graduation rates for Hispanic/Latino students than Cohort 4. 
Multilevel analyses controlling for school- and student-level variables, showed that this 
difference was statistically significant. 
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Figure 11 
Changes in the Percentages of Graduating Students in Cohorts 2 and 4 
Over Four Years by Ethnicity and for LEP and SWD Subgroups 
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Did suspension rates and the percentage of students suspended in Cohort 2 decline overall 
and by subgroup compared to Cohort 4 by the end of Year 2 of MTSS Implementation? 


In 2017-18, Cohort 2 suspension rates did not show any patterns of 
decline and remained the highest for Black/African American 
students and students with disabilities in both Cohorts. There was no 
significant difference between the Cohorts in suspension rates. This 
finding is not surprising because of the limited focus on 
implementation and no professional learning focus on behavior. 


Suspension Rates 


Since MTSS implementation was mostly focused on the Academics component and specifically targeted 
strengthening Tier 1 core instruction, it is probably too early to anticipate significant changes in student 
behavioral outcomes. Current suspension rates essentially should be considered baseline data, since 
Cohort 2 was not specifically trained on implementing any strategies to decrease suspension rates. Not 
surprisingly, by 2017-18, Cohort 2 suspension rates (number of suspension incidents per 100 students) 
did not show any patterns of decline attributable to MTSS implementation. Suspension rates remained 
the highest for Black/African American students and students with disabilities for both Cohorts. In 2017- 
18, there was no significant difference between the two Cohorts. 


Table 7 
Cohort 2 and 4 Suspension Rates (Number of Suspension Incidents per 100 Students) 
by Ethnic Group, LEP, and SWD Status, from 2014-15 to 2017-18 


Cohort 2 Cohort 4 
change change 
from from 
2014-15 2015-16 2016-17 2017-18 2014-15 |2014-15 2015-16 2016-17 2017-18 2014-15 

All 5.2 5.7 5.7 59 () o7] 51 5.8 5.4 5.4 0.3 
Black/Afr American 15.9 16.3 16.6 17.1 (9) 12) 15.7 17.9 16.6 15.8 0.1 
White 1.8 2.2 2.4 24 WW) ose 15 1.7 1.7 2.0 0.5 
Hispanic/Latino 5.3 7.1 6.0 69 [Mae 41 4.9 5.1 5.1 1.0 
Other 5.2 6.6 7.7 55 fl 0.3) 5.5 6.8 5.8 6.4 0.9 
Asian 0.4 0.5 0.6 os ff 0.4) 1.0 0.7 0.9 1.0 0.0 
LEP 5.0 6.1 5.5 59 [) o9 47 4.7 4.6 3.7 -1.0 
SWD 19.7 20.8 22.8 21.9 Pa 175 21.3 18.3 20.5 
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Percentage of Unique Students Suspended 


The percentages of Black/African American and SWD students who were suspended remained high in 
both Cohorts. As of 2017-18, MTSS professional learning had not focused on this component, so the 
results should only be considered a baseline. 


Table 8 
2014-15 to 2017-18 Percentages of Cohort 2 and 4 Students Who Were 
Suspended by Ethnic Subgroup, LEP, and SWD Status 


Cohort 2 Cohort 4 
change change 
from from 
2014-15 2015-16 2016-17 2017-18 2014-15 |2014-15 2015-16 2016-17 2017-18 2014-15 
All 3.0% 3.4% 3.4% 3.4% I) 04%] 3.3% 3.5% 3.3% 3.2% 1% 
Black/Afr American 8.4% 8.9% 9.1% 9.5% WHS 9.3% 9.9% 9.3% 8.7% 6% 
White 1.2% 1.5% 1.6% 1.5% [0.3%] 1.2% 1.2% 1.2% 1.4% 0/20% 
Hispanic/Latino 3.5% 4.3% 4.0% 4.5% WS HO%| 3.2% 3.2% 3.5% 3.4% % 
Other 3.3% 4.2% 3.8% 3.5% ff 0.2%] 3.2% 4.6% 4.1% 3.9% 
Asian 0.4% 0.4% 0.5% 0.6% 0.0%] 0.9% 0.5% 0.7% 0.7% 2% 
LEP 3.1% 3.7% 3.7% 4.0% f0Is%| 3.5% 2.8% 3.1% 2.6% 9% 
SWD 9.6% 10.2% 10.6% 10.7% WG%) 95% 10.1% 8.8% 9.5% .0% 


Did the percentage of Cohort 2 students eligible for special education support in high 
incidence disabilities categories become more representative of the population from 2014-15 


to 2017-18 and compared to Cohort 4? 


Across years a significant reduction in eligibility for special education support 
in high incidence disability categories was only observed among Cohort 2 
Hispanic/Latino students at elementary school level. Overall eligibility rates 
for special education services were similar for Cohorts 2 and 4. Black/African 
American and Hispanic/Latino students remained overrepresented in most 
high incidence disability categories. Since implementation in this area is still 
unfolding, these results should be considered a baseline. 
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Within the MTSS framework, special education services are provided to students who have a 
disability, do not respond to intervention, and have been determined to require specially designed 
instruction provided through an Individualized Education Program (IEP). Only students who have 
received sufficient academic support using evidence-based strategies and still did not show 
satisfactory progress, should be considered eligible for special education services. This is especially 
applicable to high incidence disabilities categories, which include Specific Learning Disability (SLD), 
Emotional Disability (ED), Intellectual Disability (ID), Traumatic Brain Injury (TBI), and Other Health 
Impairment (OHI). 


The long-term goal set for MTSS includes the reduction of the percentages of certain subgroups of 
students with high incidence disabilities so they become more representative of the larger student 
population after three to four years of implementation of MTSS. Implementation of this particular 
aspect of MTSS in the district is still in the early stages in terms of structures, practices, and 
professional learning. 


Over time, the proportions of students eligible for special education support in both cohorts 
decreased slightly; and the reduction over time was significant for Cohort 2 Hispanic/Latino students 
at elementary school level only, based on odds ratio analysis (although the overall percentage change 
was no different from Cohort 4). However, at all school levels, eligibility rates for special education 
services in high incidence disability categories among Black/African American and Hispanic/Latino 
students remained higher than the eligibility rates of White students. As an example, in 2017-18, in 
Cohort 2, at the elementary school level, 10% of Black/African American students were eligible for 
special education support compared to 5% of White students (see Table 9). This meant that in 
Cohort 2, the probability of being placed for special education services in the high incidence 
disabilities category for Black/African American students was twice as high as for White students. 


Table 9 
2014-15 to 2017-18 Comparisons of Cohort 2 Percentages of Elementary, Middle, and High School 
Students Eligible for Special Education Services in High Incidence Disabilities Categories 


ELEMENTARY 
Cohort 2 Cohort 4 
change from change from 
2014-15 to 2014-15 to 
2014-15 2015-16 2016-17 2017-18 2017-18 2014-15 2015-16 2016-17 2017-18 2017-18 
White 6.1% 5.7% 5.5% 5.1% [hon 5.9% 5.5% 4.8% 4.3% | -1.6% 


Hispanic/Latino 9.5% 9.1% 9.1% 7.9% Ec 9.3% 9.9% 8.8% 7.7% | 1.6% 
Black/AfrAmer 11.2% 10.5% 11.1% 10.3%  [MMMBo9%) 124% 115% 10.6% 9.5% ES 
Asian 1.5% 1.4% 1.2% 1.0% Ho5% 2.1% 2.0% 1.8% 2.1% 0.0% 
Other 6.5% 8.6% 8.9% 7.4% 65%] 7.4% 3.6% 8.2% 6.6% -0f8% 


Table continued at next page 
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Table 9 (continued) 
2014-15 to 2017-18 Comparisons of Cohort 2 Percentages of Elementary, Middle, and High School 
Students Eligible for Special Education Services in High Incidence Categories 


MIDDLE 
Cohort 2 Cohort 4 
change from change from 
2014-15 to 2014-15 to 
2014-15 2015-16 2016-17 2017-18 2017-18 2014-15 2015-16 2016-17 2017-18 2017-18 
White 8.4% 8.0% 7.1% 6.4% 9.0% 8.6% 8.3% 7.9% 


14.3% 14.2% 14.7% 14.7% 
19.9% 18.9% 17.0% 16.4% 

2.1% 1.7% 2.0% 1.5% 
11.3% 10.0% 10.4% 11.8% 


Hispanic/Latino 12.8% 13.3% 13.0% 13.3% 
Black/Afr Amer 18.6% 17.5% 17.7% 16.6% 
Asian 2.3% 2.1% 2.6% 2.2% 
Other 10.2% 9.5% 8.2% 7.2% 


HIGH 


Cohort 2 Cohort 4 
change from 
2014-15 to 


2014-15 2015-16 2016-17 2017-18 2017-18 


3.0% 8.0% 7.0% 6.8% Er) 


change from 
2014-15 to 
2014-15 2015-16 2016-17 2017-18 2017-18 


White 8.6% 8.1% 7.3% 6.8% ay 


Hispanic/Latino 12.2% 109% 10.8% 11.2% 9.6% 10.2% 9.7% 8.8% 018%) 
Black/AfrAmer 19.9% 17.9% 17.4% 18.0% 18.1% 168% 184% 168% [ey 
Asian 2.0% 2.2% 1.9% 2.1% 0.19 1.8% 2.1% 1.9% 1.6% -0.2% 
Other 11.1% 118% 111% 10.9% -0.2 11.6% 105% 10.4% 9.6% 7 
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As of 2017-18, student outcomes reflected varying levels of implementation of MTSS components. 
While more emphasis was placed on some critical components, others had not yet been implemented 
at the time of this report. 


As of 2017-18, implementation of MTSS did not show any significant impacts on long-term student 
outcomes, except for the improved graduation rates for the Cohort 2 Hispanic/Latino subgroup 
compared to Cohort 4. According to Implementation Science, the time it takes for an innovation to 
move from initial to full implementation will vary depending upon the complexity of the new program 
or innovation, as well as the development of the infrastructure to support teachers and the availability 
of implementation supports and resources (National Implementation Research Network, n.d.). 
Research also shows that it generally requires three to five years of implementation with fidelity for the 
outcomes to show (Fixsen et al., 2005). 


In WCPSS, the MTSS framework initial implementation is still unfolding. Much has been accomplished, 
but much more still needs to be done to move to the full implementation stage such that longer-term 
student outcomes should begin to be positively affected. Since MTSS is a complex framework with 
many components, it takes time to fully implement all six critical components. Thus, it will be important 
for the district to “stay the course” to ensure that MTSS can be implemented with fidelity. 


As of fall of 2018, the School Improvement Plan (SIP) redesign team has finished aligning the school 
improvement process with the MTSS framework. A Cross-Departmental Behavior Team developed and 
presented the Tiered Behavior Resource (TBR) to elementary school principals and planned to share it 
with middle and high school principals some time in 2018-19. 


A Standard Treatment Protocol (i.e., Tier Il and Ill Intervention Systems and Structures) for literacy, 
math, behavior, and attendance is being developed. The district is currently using an Early Warninng 
System (EWS) for monitoring response to instruction in order to develop academic and behavior plans 
and will transition to the state’s system in summer 2019. The district is also aligning systems and 
structures of support by creating tiers of supports for schools from Central Services. As more and more 
of these components fall into place, the district’s ability to examine the effects of MTSS will increase. 


Recommendations 


Recommendations focus on ways to strengthen further MTSS implementation through Central 
Services support and inter-departmental collaboration: 


e The MTSS framework should remain a district priority. It is still being developed and is at initial 
stages of implementation on some dimensions. The district needs to maintain consistent 
leadership team and inter-departmental support and collaboration at all levels, to provide effective 
assistance to school leadership teams, school counselors, and intervention specialists. 
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e Early Warning System (EWS) data should be used more consistently to identify the schools with 
more barriers to implementation of MTSS and offer those schools additional implementation 
support. 


e Additional emphasis on the behavioral component of MTSS is needed through district-level cross- 
departmental collaboration to improve implementation fidelity for behaviorally focused MTSS 
strategies; the adoption of a K-12 behavior assessment system will be helpful to further this work. 


e School Improvement Plans have been restructured as a result of a year-long effort of the district’s 
MTSS/SIP leadership team. District leadership should continue to monitor and strengthen 
implementation of the new school improvement process to maintain alignment with MTSS. 


e Strengthen application of standardized procedures for tiered student support in Tiers II and II| and 
more appropriate identification of students eligible for special education support in high incidence 
disability categories. Continued support and learning are needed to help schools prepare for the 
2020 Specific Learning Disability (SLD) Eligibility policy. 


e Since the MTSS logic model is a living document, the district leadership should annually revisit and 
update the document to reflect implementation progress. 
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Appendix A 


Figure Al 
Implementation Stages 
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Source: National Implementation Research Network (NIRN). 


e The exploration stage involves acquiring information about the innovation and creating awareness 
about an innovation, identifying the district needs, assessment of the fit, exploring resources, and 
assessing the readiness to replicate. In 2014-15, the researchers from the Florida Problem 
Solving/Response to Intervention Project housed at the Florida Department of Education and the 
University of South Florida, introduced the MTSS framework to the district and trained the district 
senior leadership team, outlined the content of professional learning for the schools, and helped 
select resources to measure implementation progress. For Cohort 2 most exploration took place in 
2015-16, which was also their first training year. 


e The installation stage includes ensuring funding for the innovation, hiring staff, policy development, 
ensuring other structural support, and defining expectations of the outcomes. At the installation 
stage, the MTSS district leadership team designed the district implementation process including 
professional learning, created a planning tool (the MTSS logic model), utilized NCDPI support in 
measuring the district capacity to implement the framework by completing the District Capacity 
Assessment, and identified the tools for implementing tiered support. An Early Warning System was 
introduced to MTSS trained Cohorts. For Cohort 2, 2016-17 became both an installation and the 
initial implementation year with continued professional learning. 


e Initial Implementation, or the “initial awkward stage,” creates changes in organizational capacity, 
culture, and skill levels. At that stage, the MTSS cohorts which received their professional learning 
examined their staff beliefs about learning and teaching, conducted self-assessment of MTSS 
implementation, and outlined their school-specific action plans. For Cohort 2, 2017-18 included 
continued professional learning and became the second initial implementation year. Analysis of the 
Self-Assessment of MTSS (SAM) completed by Cohort 2 schools before and after a year of 
implementation showed increased readiness and overall increased levels of implementation of the 
six MTSS components. However, it is important to note that the significant increase that took place 
moved the schools from “not implementing” to “emerging/developing” levels; some schools 
reached the “operationalizing” level, but none reached “optimizing” levels by the end of 2017-18. 


30 


MTSS 2016-17 to 2017-18 December 2018 


e Full Implementation is based on new learning being integrated into daily practice, a framework 
becoming fully operational (National Implementation Research Network, n.d.). At this stage, schools 
should start viewing new practices as “a routine or accepted practice,” and staff gain skills to 
implement the initiative (Fixsen et al., 2010). 
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Appendix B 


Multilevel Analyses 


In addition to descriptive analyses conducted throughout, we conducted a series of multilevel analyses 
to control for possible effects of prior student achievement, student demographic characteristics, and 
school effects. Such statistical control allowed for a more precise measurement of outcomes and gave 
us the ability to attribute any positive results to the implementation of MTSS, even when the treatment 
effects were small. 


To explore whether MTSS had any differential effects on students, we included cross-level interactions 
between the school-level predictor (Cohort 2 vs. Cohort 4) and student-level predictors, such as Limited 
English Proficient (LEP) status and special education (SWD) status in our multilevel model. 


The one statistically significant result detected was related to the graduation rates for Hispanic/Latino 
students. The odds for Hispanic/Latino students in Cohort 2 high schools to graduate was 73% higher 
(odds ratio: 1.73) than Hispanic/Latino students in Cohort 4 high schools (95 Confidence Limits (CL): 
1.30, 2.32) keeping other factors the same. 


A representative model showing only the subgroup of Hispanic/Latino students is provided below for 
illustrative purposes. The school random and student error are identified by the following two terms, u, 
J] 
and e_in the equation, respectively. The coefficient yo: represents the Cohort’s main impact and the 
y 


coefficient, yi1represents the Cohort’s impact on the ethnic subgroups, which is the focus of this 
subgroup analysis. 


Bis. x é ; , i 
In( oe ara d sot lispanic,, + Y Cohort, +y Hispanic Cohort, +u +e. 


1- 0j ij 


where i=i' student 
j=j' school and 
p; = is the probability of the ith student graduating. 


32 


